JULY  1,  1979 


OPTIMAL  REPLACEMENT  OF  PARTS  HAVING  OBSERVABLE 
CORRELATED  STAGES  OF  DETERIORATION 


by  L.  Shaw,  C.  L.  Hsu  and  S.  G.  Tyan 


POLY-EE-79-055 


Program  in 

SYSTEM  AND  DEVICE  RELIABILITY 


Contract  No.  N000 14- 75 -#-085 8 


79  07  23  06  2 


JULY  1,  1979 


OPTIMAL  REPLACEMENT  OF  PARTS  HAVING  OBSERVABLE 
CORRELATED  STAGES  OF  DETERIORATION 


by  L.  Shaw,  C.  L.  Hsu  and  S.  G.  Tyan 


POLY -EE-79-055 


Program  in 

SYSTEM  AND  DEVICE  RELIABILITY 


Contract  No.  N000 14-75-6-0858 
Office  of  Naval  Research 


i 


ill 

J 


Unclassified 

security  classiricatiqn  or  this  rage  fi»i»n  o»i«  gwcuo 

REPORT  DOCUMENTATION  PAGE  befoIeDcompletwoNfoRm 

' T2J55SS35-5**— J jToHTTIccetlTSrSa  ^^iorient-s  catalog  numun 

) 4"  ]POLY-EE^79-)j45-J  I fq) 

^ *===?^?  |rtrf«‘  * - V <?r  "•™T  * ■ 

^ Optimal  Replacement  of  Parts  Having  Observable  j T'rec}injca^  x'^Xr-V- 

*‘w’  Correlated  Stages  of  Deterioration  . 7 J ^ ! 

! t RtREORMING  OKG.  RERORT  NUMBER 

1.  AUTHfl^.l  ■ --  «•  ^fluTiiCT  a»  CAM T 


E7  AUThORCJ 

L./Shaw,  C.  L.l  Hsu. 


>.  G.^Tyan 


1^614-75-0^? 


I*,  rerrorming  organiIation  name  ano  aodress 


Polytechnic  Institute  of  New  York 

333  Jay  Street  J>^7  '4-  £»«<-<  £>J-  -4 

Brooklyn,  New  York  11201  £ I c c.  -Tv  ,;p  K.^s.1  c . 


5.  PROGRAM  EL  EM  EN  T.  RROJECT.  Task 
AREA  * WORK  UNIT  NUMSERS 


" CONTROLLING  ORRlCE  NAME  ANO  ADDRESS  f 

Chief  Naval  Research  (Code  436)  f 

800  N.  Quincy  ^ 

Arlington.  Virginia  22203 

'*•  MONITORING  AGENCY  name  5AOORE5S(-||  Jlitjtfmt  lrom  Controil^g^ 

Office  of  Naval  Research  ""T 

207  W.  24th  Street  f /O  JDQn 

New  York,  New  York  10011  • ' r 


I l«.  DISTRIBUTION  statement  !ot  thl.  Report) 


llli 


JulfB»7  9 


»J  15.  SECURITY  CLASS,  (ai  :h it  rtport) 

I Unclassified 


IS«.  DECLASSIFICATION/  DOWNGRADING 
SCHEDULE 


Reproduction,  in  whole  or  in  part  is  permitted  for  any  purpose 
of  the  United  States  Government. 


• ?.  DISTRIBUTION  STATEmCNT  (‘of  th e ebatrect  anterad  in  3f>cfc  20.  II  different  from  R#|*®rt> 


[ »•.  SURRLfiMCNTARY  NOTES 


tt.  KtY  *ORO$  fConttn ue  on  nvs««  aid*  if  neceeaerv  «nd  Identity  by  block  number) 

Multivariate  Exponential  Distribution,  Optimal 
Replacement,  Aging,  Deterioration 


20  ABSTRACT  'Continue  *»  reeeree  ride  II  neceeeery  end  Identity  bv  block  number) 

SEE  NEXT  PAGE 


Vo  1 71 


00  I JANETS  1473  EOlTION  OR  ' NOV  S3  IS  OBSOLETE 


ICol 


Unclassified 

SECURITY  CL  ASSlRI  CATION  OR  This  RAG«  W««o  Data  Em.r., t) 


Unclassified 


»«CU*lTr  CLASSIFICATION  OF  THU  PAQgflWiwi  Oara  fAfararfj 


ABSTRACT 

*A  single  component  system  is  assumed  to  progress  through  a finite 
number  of  increasingly  bad  levels  of  deterioration.  The  system  with  level  i 
(0f<> i < n)  starts  in  state  0 when  new,  and  is  definitely  replaced  upon  reach- 
iri$:,fcK‘e"worthless  state  n.  It  is  assumed  that  the  transition  times  are  directly| 
monitored  and  the  admissible  class  of  strategies  allows  substitution  of  a 
new  component  only  at  such  transition  times.  The  durations  in  various 
deterioration  levels  are  dependent  random  variables  with  exponential 
marginal  distributions  and  a particularly  convenient  joint  distribution. 
Strategies  are  chosen  to  maximize  the  average  rewards  per  unit  time.  For 
some  reward  functions  (with  the  reward  rate  depending  on  the  state  and  the 
duration  in  thi3  state)  the  knowledge  of  previous  state  duration  provides 
useful  information  about  the  rate  of  deterioration. 


T 


“Accession  For 
GSA&I 
DOC  TAB 

Unannounced  £ LJ  I 

J os  t i f i c n 1 1 0 n~T^  ( ' ’ 

I pistribuilSS/- 

" | Avail  and/or 
,T))st  I special 


ft 


jicu*ity  classification  Of  this  •»aec>wi«n  3«*»  Emwh) 


s 


i 


OPTIMAL  REPLACEMENT  OF  PARTS  HAVING  OBSERVABLE 
CORRELATED  STAGES  OF  DETERIORATION* 

by 

L.  Shaw^,  C-L.  Hsu  , S.  G.  Tyan^ 

ABSTRACT 

A single  component  system  Is  assumed  to  progress  through  a 
finite  number  of  increasingly  bad  levels  of  deterioration.  The  system 
with  level  i (0  < i < n)  starts  in  state  0 when  new,  and  is  definitely  re- 
placed upon  reaching  the  worthless  state  n.  It  is  assumed  that  the 
transition  times  are  directly  monitored  and  the  admissible  class  of 
strategies  allows  substitution  of  a new  component  only  at  such  transi- 
tion times.  The  durations  in  various  deterioration  levels  are  depen- 
dent random  variables  with  exponential  marginal  distributions  and  a 
particularly  convenient  joint  distribution.  Strategies  are  chosen  to 
maximize  the  average  rewards  per  unit  time.  For  some  reward  func- 
tions (with  the  reward  rate  depending  on  the  state  and  the  duration  in 
this  state)  the  knowledge  of  previous  state  duration  provides  useful 
information  about  the  rate  of  deterioration. 
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Many  authors  have  studied  optimal  replacement  rules  for  parts 
characterised  by  Markovian  deterioration,  for  example  Kao  [6]  and 
Luss  [ 9]  and  the  many  references  found  in  those  papers.  Kao  mini- 
mized the  expected  average  cost  per  unit  time  for  semi-Markovian 
deteriorating  system,  and  considered  various  combinations  of  state  and 
age-dependent  replacement  rules. 

Luss  examined  inspection  and  repair  models,  where  he  assumed 
that  the  operating  costs  occurring  during  the  system's  life  increase  with 
the  increasing  deterioration.  The  holding  times  in  the  various  states 
were  independently,  identically,  and  exponentially  distributed.  The 
policies  examined  include  the  scheduling  of  the  next  inspections  (when  an 
inspection  reveals  that  the  state  of  the  system  is  better  than  certain 
critical  state  k)  and  preventive  repairs  (when  an  inspection  reveals  the 
state  of  the  system  being  worse  than  or  equal  to  k).  The  convenience  of 
a Poisson-type  structure  for  the  number  of  events -per -unit -time  made 
it  relatively  easy  to  allow  general  freedom  in  the  selection  of  observa- 
tion times. 

The  work  studied  here  is  based  on  a modification  of  the  model 
used  by  Luss.  Our  model  for  deterioration  is  more  general,  but  the 
admissible  strategies  used  here  are  more  restricted.  Here  we  allow 
the  exponentially  distributed  durations  to  have  different  mean  values, 
and  to  be  positively  correlated. 

The  introduction  here  of  correlation  between  interval  durations 
permits  the  modeling  of  a rate  of  deterioration  which  can  be  estimated 
from  a particular  realization  of  the  past  durations.  However,  the  lack 
of  a Poisson-type  of  structure  for  the  events -per -unit -time  makes  it 


much  more  difficult  here  to  allow  general  freedom  in  the  selection  of 
observation  times.  At  present,  only  the  simple  case  of  direct  and 
instantaneous  observation  of  deterioration  jumps  has  been  considered. 

This  model  would  be  appropriate,  for  example,  in  a subsystem 
which  functions,  but  with  reduced  efficiency,  when  some  redundant 
components  have  failed;  and  for  which  failure  of  one  component  might 
indicate  environmental  stresses  which  increase  the  probability  of  fail* 
ure  for  other  components.  In  addition,  deterioration  in  correlated 
stages  might  be  used  as  a simple  approximation  for  a continuously 
varying  degradation  which  does  not  exhibit  discrete  stages. 

Figure  1 shows  a typical  time  history  of  deterioration  and  replace- 
ment. The  duration  in  state  (i-i),  prior  to  reaching  state  (i),  is  r.  ^ . 

The  intervals  d^  in  Figure  1 represent  the  time  required  to  replace  a 
component  when  it  has  entered  state  i.  The  sequence  {r^}  will  be  Markov, 
characterized  by  a multi-variate  exponential  distribution.  Reward  func- 
tions will  be  related  to  the  deterioration  state  and  the  time  spent  in  each 
state.  The  decision  rule  specifies  whether  or  not  to  replace  when  enter- 
ing each  state  i,  on  the  basis  of  the  history  of  r.  r.  ^ . The 

Markov  property  simplifies  the  decision  rule  to  be  a collection  of 
sets  such  that  we  replace  on  entering  state  i if  and  only  if  r.  ^ e (?.. 

The  objective  is  to  maximize  the  average  reward  per  unit  time: 


(Total  reward  in  (0,  T )) 


Ef  Reward  per  renewal! 

E[  Duration  between  renewals 


(See  Ross  [ 1 1 ] page  160  for  equivalence  of  (1 ) and  (2).  ) The  mean 
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reward  per  renewal  is  defined  here  as: 

(3) 

in  which: 

N * state  at  which  replacement  occurs  (possibly  random), 

I*N  s replacement  cost  if  replaced  on  entering  state  N 
(possibly  random), 

c^t)  a reward  rate  when  in  state  i. 

Figure  2 shows  several  reward  rate  time  functions  c(t)  which 
have  been  considered.  When  one  of  these  c(t)  functions  is  specified 
for  a given  problem,  the  c^t)  in  (3)  are  assigned  values  8jC(t)  with: 


& * E 


N-i  ri 

Z / Cjltwt  - pN 


i*0 


> Ja.,  i J, 


(4) 


to  assure  greater  reward  rates  in  less  deteriorated  states.  State  n 
corresponds  to  a completely  failed  or  worthless  component. 

The  mean  duration  in  (2)  is  defined  as: 


4*  E 


N-l 

S ri  + dN  • 
iaO  1 N 


(5) 


to  include  a possibly  random  time  d^  for  carrying  out  a replacement  at 
state  N. 

While  the  ultimate  objective  is  to  choose  C-^  to  maximize  the  L de- 
fined in  (i),  it  is  well  known  that  a related  problem  of  maximizing: 


£q(o)  3 /?-  a D, 


(b) 


is  simpler  [ l].  Indeed,  the  C . which  maximize  L.  will  be  identical  to 


those  which  maximise  £^(q ) for  the  a*  such  that: 


£jj(ar*)  * 0,  where  £g(o)  £ max  £^(a) 


ic.) 


Section  1 considers  a case  in  which  it  is  found  that  deterioration 
rate  information  is  not  useful  (e.  g. , the  optimal  policy  is  Independent  of 
the  amount  of  correlation  between  successive  state  durations). 

Sections  2 and  3 consider  other  reward  rate  structures,  e.  g. , as- 
suming that  more  deteriorated  parts  are  rustier,  hotter,  or  more  brittle, 
and  therefore  more  costly  to  replace.  In  such  cases  the  optimal  policies 
do  make  use  of  estimates  of  the  deterioration  rates  as  well  as  of  obser- 
vations of  the  deterioration  level. 

The  appendix  describes  useful  properties  of  the  multivariate  ex- 
ponential {r.}  sequence  which  is  used  to  model  the  correlated  residence 
times  in  a sequence  of  deterioration  3tates. 

1.  Constant  Reward  Rate  - State  Independent  Replacement  Penalties 
The  constant  reward  rate  case  with  c.(t)  = 3^  and  with  state- 
independent  replacement  penalties  (p.  * p,  d.  = d)  is  particularly  simple 
to  analyze.  We  will  see  that  as  long  as  E[r.|  r.  . , r.  . . . ] > 0 for  all 
i,  even  if  the  r^  are  not  exponentially  distributed,  the  optimal  rule  will 
be  to  replace  the  deteriorating  part  upon  entering  some  critical  state  k*, 
independent  of  the  observed  durations  r.. 

Based  on  the  problem  statement,  the  optimal  decision  on  entering 
state  j must  maximize  the  mean  future  reward  until  the  next  renewal, 
£.(cr),  for  a suitable  a.  Here; 
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£.(<*) 


• * 

N-1 

■ • 

N-1 

E v,h-, 

• m 

-Or£ 

2 r.|r. 

i«j  1 J’1 

-p-ad. 


(8) 


Immediately  after  a renewal,  when  j * 0,  the  expectations  defining  £g(«) 
are  unconditional.  The  optimal  decisions  for  each  state  will  be  found  in 
terms  of  or,  and  then  the  proper  a*  (for  producing  decisions  which  maxi* 
mize  L)  is  the  one  for  which  the  maximum: 


max  (or *• ) * * 0. 


(9) 


Optimization  by  dynamic  programming  begins  by  considering  the 
decisions  at  the  last  step,  i.  e. , on  entering  state  (n-1 ).  There  are  two 
choices,  to  replace  (R)  or  not  to  replace  (IT),  with  corresponding  values: 


iQ_j  (a  :R)  = -p-ad,  (10) 

and: 

£n-lta;5’1*  Eti3n-lrn-l|tn-21  ' aE^  rn-l 1 "n-21  'I""1 

* E(  ,Sn-t  •“  ,rn-l  I 'n-21  -p-“d‘  ' “ 1 

Clearly,  the  best  decision  is  not  to  replace  if  and  only  if  the  difference 

A«-l(a;rn-*,A  -*».!  <“  *> 

- (Sn.,<.)  E(rn.,|rB_2]  > °.  (12) 


is  non-negative.  The  sign  of  (12)  will  be  the  sign  of  (/3r  ^ -or),  due  to  the 
non-negativity  of  all  interval  durations.  Thus  the  best  decision  depends 
on  a and  the  reward  parameter  ^ , but  not  on  the  previously  observed 
duration.  Two  cases  will  be  considered  separately. 
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if  & , > or  than  the  best  decision  at  state  (n-l ) is  not  to  replace, 

n-l  — 

We  will  now  explain  why,  under  this  condition,  it  is  best  not  to  replace 
at  any  state  less  than  n.  Consider  the  situation  on  entering  (n-2).  We 
have  already  shown  that  it  is  better  not  to  replace  on  entering  (n-l). 
Thus  the  choice  will  be  based  on  a ^ n_2  °*  the  form: 


Here  we  have: 


>0- 


by  assumption,  and: 

EtVl|rn-jli°  *"d  Etr„-2I  ra-J>  i °-  (,5) 

because  all  r,  > 0 with  probability  one.  Thus  ^n_2^a  :rn-3^  > ® f°r 
rn  3 > 0,  and  it  is  also  better  not  to  replace  here.  This  argument  can 

be  repeated  for  states  (n-3),  (n-4) 1,  0. 

The  other  case  to  consider  is  3n_j  < <*»  which  requires  replace- 
ment on  entering  state  (n-l ),  if  the  system  ever  reaches  that  state. 

When  we  consider  the  decision  on  entering  (n-2),  the  Aq_2  *-3: 

An-2(a  :rn-3)a  Et(0n-2'a)rn-2l  rn-3] * (io) 

which  has  the  sign  of  (£n_2-cr).  U (^n-2*Cf)  < °’  th#n  r®PUcement  is 
optimal  on  entering  (n-2)  and  (n-3)  is  considered  next.  This  iteration 
may  eventually  reach  a state  (k-l)  where  -or)  > 0 and  it  is  better 

not  to  replace.  Arguments  similar  to  those  for  the  -<*  > 0 case 
show  that  non-replacement  is  the  optimal  decision  at  all  state  preceding 
the  one  which  first  arises  as  a non-replacement  state  in  this  backward 


—7" 


-8- 


iteration. 

In  summary,  in  the  constant  reward  rate-constant  replacement 
penalty  case  -Cq(o)  is  maximized  by  a decision  rule  which  says  replace 
on  entering  some  state  k ^ n which  depends  on  the  reward  parameters 
{3^}  and  the  at : 

k=  min{i:(or-3i)  > 0}  . (i7) 

Finally,  we  must  choose  a*  so  that  £g(a*)  = 0,  where: 

0 k“l 

r.Q(or)  = -p-od  + 2 (3.-a)E[r  ] (18) 

0 1 1 

Figure  3 shows  a typical  plot  of  j^(a)  as  a continuous,  piecewise 
linear  curve  whose  zero  crossing  (J^(a-*)  = 0)  defines  or*  and  the  opti- 
mal replacement  state  k*  for  maximizing  L». • 

Example:  Figure  3 shows  that  the  optimal  average  reward  per  unit  time 
is  L = 2 i when  k*  = 3,  where  3Q  = 5,  = 4,  30  = 3,  33  = 2,  34  = 1, 

3g  = 0,  p = 5,  d = 1,  = 2 (i  = 0,  1,2,  3,  4)  and  n = 5.  F rom  Equation 

(18),  the  optimal  k is  a function  of  or,  which  remains  constant  when  at 
varies  over  each  interval  3 < at  < 3^.  as  shown  in  the  figure.  *■ 

2.  Increasing  Replacement  Penalties -Constant  Reward  Rate 

Here  we  generalize  the  model  of  the  previous  section  by  allowing 
the  replacement  cost  p^  and  replacement  duration  d^  to  be  functions 
of  the  replacement  state  (i),  and  to  be  random.  These  parameters  are 
assumed  to  have  mean  values  E(  p^J  and  E(d.J  which  are  convex  non- 
decreasing sequences  in  i,  corresponding  to  the  increased  difficulty 
in  replacing  more  deteriorated  parts  which  may  be  e.  g. , rustier, 


-9- 


hotter  or  more  brittle.  We  also  assume  that  the  mean  durations  are 

ordered:  v > rj  > . . . > rj  , , corresponding  to  faster  transitions  of 
o — 1 — — n-i 

more  deteriorated  parts. 

The  foregoing  assumptions,  together  with  properties  of  the  as- 
sumed multivariate  exponential  density  for  stage -durations  (see  Appen- 
dix), lead  to  an  optimal  decision  policy  with  a nice  structure.  That 
optimal  policy  prescribes  replacement  when  entering  state  j,  if  and 
only  if  r.  ^ < r?  where  the  decision  thresholds  are  ordered: 

»<  'o/V*/,,l  £•••  -rn-l/nn-l  * " • 

The  optimal  decision  on  entering  state  j must  maximize  the  mean 

future  reward  until  the  next  renewal,  i.  e. , £.  (a).  For  a suitable  or,  we 

have: 


£..(0f)=E|Z  p.r.lr.^j  -aEjz  r.|  r.  { j -E[  pN  + adN]  (19) 

For  notational  simplicity  we  define  e^  = E[p^  + ad^]  and  note  that  e^  is 
also  convex  and  non-decreasing  since  we  are  only  interested  in  a > 0. 
The  optimal  decisions  for  each  state  will  be  found  in  terms  of  a,  and 
then  the  proper  a * (for  producing  decisions  which  maximize  L)  is  the 
one  for  which  the  maximum  £ vanishes: 

0 [n-1  N-i 

£n(a*)  * eM(a*)  + E 2 0,  r.  - a*  E r.  = 0 . (20) 

0 N i=0  1 i=0  1 

- 

Optimization  by  dynamic  programming  begins  by  considering  the 
decision  at  the  last  step.  Since  state  n represents  a failed  component, 
we  definitely  replace  the  component  when  it  enters  state  n.  Next,  we 
consider  the  decision  to  be  made  on  entering  state  n-i.  There  are  two 


choice*:  to  replace  (R)  or  not  to  replace  (IT),  with  cor  responding 


values 

£ i <«  :R)  * , * 

n-i  n-i 


(21) 


E[fln-lri>-l  '“Vl1'..!1  -*n  (22> 

(or  £.  (a).  Clearly,  the  best  decision  is  not  to  replace  if  and  onlv  if 

n-l 

£Vl,rn-2)A£n-l(“;ff)  -£„-lta>RI 

is  non-negative,  i.  e. , . 


Vl(V2>-  E[  rn-l 1 rn-3J  * 'Vl^i 


0. 


(23) 


Referring  to  (A-o),  ^,.^rn_2^  l*  * linear  function  of  rn_.y  with 


Vl 101  ’ ,Bn-l  -“'"n-t 11  •» 1 * <*o-l 

Figure  4 shows  the  possible  shapes  for  this  function.  There  can  be  no 

downward  zero-crossing  at  an  r „ > 0. 

n-  j 

Thus,  depending  on  the  numerical  values  of  the  parameters,  there 
are  three  possible  kinds  of  optimal  decision  rules  when  entering  state 
(n-l ): 

l)  replace  for  any  r 0 if  A > 0 for  all  r ^ > 0 

n*«  n-i  * n- - — 

il)  replace  for  no  r if  A , < 0 for  all  r 0 * 0 

n-J  n-l  ~ n - - — 

iii)  replace  if  and  only  if  r^„  > rn.  >—  wh*r®  A^_t  irn_2^  * °* 


In  other  words, 


^n-l(tf) 


{rn-2*  rn-2<rn-2}* 


(24) 
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where  r „ could  be  aero  (case  ll)  or  Infinite  (case  i). 
n-  « 

Next  we  consider  the  optimal  decision  when  entering  state  (n-2), 
and  assuming  that  the  optimal  decision  will  be  made  at  the  subsequent 
stage.  vVe consider  cases  of  K ®)  »n<*  (/3n-1  - *)  •®p**’*t®ly» 

a)  < «)  Implies  replacement  on  entering  (n-1),  so 

Vi"..)1'  'Vi-*1  E(rn-2lrn-3l  + (,n-2  * *«-t >• 

resulting  in  the  same  three  possibilities  listed  above  for  state  (n-1). 

b)  for  On_t  > or): 

Ah-2(rB-J)*  S-2  + Wb-2-“' 

+ / t t»a-l  ■«>  rn-l  I rn.2l  “J  f(ra-2l  rn-3^ra-2 


♦ 


We  now  prove 

A»  WnO’iV.'Vj' 

B)  6 „(s  -j ) is  convex  with  at  most  one  upward  zero  crossing 

n-«  n - j . ^ n 

at  an  s > 0. 

There  is  no  harm  in  writing  *n.^sn_3)  or  fin-l(®)  instead  of  ^n.^sn_2^  *or 
purposes  of  comparing  functions. 

To  prove  A),  consider 

«„.2(s)  -6n-l|s,=  ♦ E[(«1>.l(*»t|  » J 

+ (•»„.!  -»l»n.2  ' Et*  *J-  <2,) 

where  s represents  the  normalized  duration  preceding  s. 

The  terms  on  the  right  side  of  (27)  are  non-negative  due  to  the 
convexity  of  the  e^,  ( )+  > 0,  (A-o),  and  the  assumed  orderings  of  the 
/3.  and  n.. 

This  completes  the  proof  that  A)  is  true.  It  follows  immediately 
that  if  i)  applies  for  state  (n-1),  then  it  is  also  optimal  not  to  replace  in 
state  (n-2)  or  any  earlier  state.  (Recall  < $n_2  < • • • . and  we 
are  now  considering  cr  < ). 

To  prove  B),  which  is  only  of  interest  when  an  rn_2  «xis*st 
we  refer  to  the  theorem  in  the  appendix.  The  test  difference  6n_2^ 
can  be  written  as 

Sn-2(*>*  eK.2  - ♦ Wn.2-a),'n.2’  * <*  n-l  (,"+l  *J  (28) 

in  which  the  integrand  has  the  properties  required  by  h(s)  in  the  theo- 

* + 

rem.  To  see  this,  we  note  that  r „ > 0 implies  that  (6„  , (0))  * 0,  so 

n- 6 n-i 
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s 


the  integrand  is  non-positive  at  s = 0.  Thus,  8n_2^8^  ^a*  *^aPe 
stated  in  B),  implying  that 

(29) 

where  r*  , may  be  zero,  infinity,  or  the  non-negative  value  defined 
n - 3 

h?  6n-3(rn-3/rTn-3)=  °* 

The  foregoing  arguments  can  be  repeated  for  rn_g»  • • rQto 

prove  that  the  optimal  replacement  policy  has  the  forms 

* 

Replace  on  entering  state  i if  and  only  if  r^<  r.^ 

where 

0 < ^ r*/nl  - •••  - r»-t/ Vl  * " 

When  repeating  the  proof  for  earlier  stages,  the  ( ) term  in  (28)  is  modi- 
fied to  the  form,  e.  g.,  [(«n.2(s))+  “ (6n-l(s))+1‘  This  term  is  Zer°  f°r 
x 9 0,  and  generally  non-negative,  due  to  A),  so  the  basic  theorem  is 

still  applicable. 

3.  Computational  Procedure 

The  preceding  section  derived  the  structure  of  the  optimal  decision 
rule  for  the  case  where  replacement  is  more  difficult  and  more  expensive 
when  the  part  is  more  deteriorated.  The  corresponding  optimal  decision 
thresholds  can  be  formed  as  follows: 

a)  choose  an  initial  a. 

b)  Find  the  r*(a)  (i  = n-2,  n-1 0)  recursively,  via  numeri- 

cal integration  of  expressions  like  (26)  (where  rQ_j(a)  is  defined  by  the 

condition  A „(r  ,)  * 0). 

n-z  n - j 


i 

l 
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c)  Compute 

00 

J^(o)  a e0-et  + J [ (0o-or)  rQ  + <^t  (rQ))+]  f(r0)drQ. 

0 

d)  If  |£n(cr)|  < c,  for  sufficiently  small  c,  say  L * a a as 
otherwise  repeat  the  computational  cycle  starting  with  a new  a. 

The  following  properties  of  £^(a)  can  be  used  to  generate  an  a- 
sequence  which  converges  to  a*. 

1.  £g(a)  is  monotone  decreasing,  since  £^(a)  has  this  property 

for  a fixed  policy  (see  Eq.  (19));  and  if  £^(ag)  > £^(0^ ) for  >0^,  then 

the  policy  used  to  achieve  £q(o,J  could  be  used  to  achieve  an 

^t)(al)  > ""  a contra<*iction. 

* 

2.  When  p = 0,  all  r^  are  zero  or  infinite:  replacement  always 

ijc 

occurs  on  arrival  at  a critical  state  i . Use  of  that  policy  will  achieve 
the  same  average  reward  for  durations  having  any  value  of  p.  Thus,  a 

>{C  ^ jj< 

useful  bound  on  a ' (p ) is  a (0)  < a (p );  0 < p < 1 . 

3.  When  p * l,  future  r^  are  completely  predictable 
(Var(rJ  r._^ ) a 0 in  (A-7),  so  a*  ( 1 ) > a*(p ).  In  this  case  there  is  es  - 
sentially  a single  random  variable  rg.  and  the  r^  can  be  calculated 
without  the  need  for  numerical  integration  of  Bessel  functions. 

4.  Numerical  Example 

Table  I lists  parameter  values  for  a replacement  problem  which 
falls  under  the  assumptions  of  Section  2. 


i 

0 

l 

2 

3 

4 

5_ 

*1 

5 

4 

3 

2 

1 

nl 

1 

0.9 

0.  8 

0.  7 

0.6 

[ Ptl 

2 

2.  2 

2.4 

2.6 

2.  8 

Ut  1 

1 

1.  1 

1.  2 

1.  3 

1.4 

Since  future  durations  are  independent  of  past  ones,  the  optimal 
policy  replaces  when  a critical  state  i is  reached.  The  general  opti 
mal  reward  expression 


Direct  evaluation  shows 


Since  r^  * r^  r?j/ rQ  in  this  case,  the  optimal  rule  specifies  a re 
placement  state  jtr^)  as  a function  for  r^. 

For  any  such  policy 


This  expectation  will  be  maximized  if  j(r.)  maximizes  the  bracketed 


term  for  each  rQ.  Making  the  neceeaary  comparisons  for  a sequence  of 
a -values  leads  to  the  policy 


for  which 


We  know  that  2.  205  < or  (-j ) < 2.  25.  A pilot  ca 
lines  indicated  in  the  previous  section  shows  that  r^( 
for  j > 2,  and 


where  a is  chosen  to  make  the  following  £q(o)  vanish. 


While  the  method  for  finding  or  (-j)  is  clear,  further  numerical  work 
seems  unwarranted.  The  precision  needed  to  get  a meaningful  answer 
is  not  justified  by  the  minimal  improvement  that  the  optimal  policy  will 
have  over  the  suboptimal  use,  for  all  p,  of  the  optimal  p = 0 policy  (re- 
place when  reaching  state  3). 
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5.  Conclusion* 

A multivariate  exponential  distribution  has  been  used  to  describe 
successive  stages  of  deterioration.  Optimal  replacement  strategies 
have  been  found  for  the  class  of  decision  rules  which  can  continuously 
observe  the  deterioration  state,  and  which  may  make  replacements  only 
at  the  times  of  state  transitions.  Similar  results  have  been  found  for  the 
other  reward  rates  shown  in  Figure  2 (linear;  and  constant  after  an  ini- 
tial set-up  interval  for  readjustment  to  the  new  state)  [?]. 

The  optimal  replacement  policy  derived  in  Section  2 makes  use  of 
observations  which  allow  estimation  of  the  current  rate  of  deterioration 
for  the  correlated  stages  of  deterioration.  The  numerical  example 
demonstrated  a situation  where  a much  simpler  policy,  which  ignores 
the  correlation,  can  be  almost  as  good.  The  relatively  easy  calculation 
of  the  optimal  rewards  for  p a 0 and  p * 1 provides  information  about 
the  necessity  for  use  of  the  more  complex  optimal  solution.  It  is  worth 
noting  that  we  have  been  unable  to  find  other  combinations  of  values  for 
the  parameters  in  Table  I such  that  there  is  a substantial  relative  in- 
crease in  the  maximum  reward  as  p increases  from  zero  to  infinity. 

The  ordering  of  state  dependent  rewards,  mean  durations,  etc. 
assumed  here  are  physically  reasonable,  and  lead  to  nice  ordering  of 
the  decision  regions.  However,  other  (3^,  ir,  p.,  d^  orderings  might 
be  more  appropriate  in  other  situations.  The  model  introduced  here  for 
dependent  stage  durations  could  be  used  in  those  cases,  together  with 
dynamic  programming  optimization,  although  the  solutions  may  not 
have  comparably  neat  structures. 

One  reasonable  generalization  would  allow  transitions  from  state  i 
to  any  state  j > i.  This  would  not  change  the  form  of  the  solution  in  the 
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Dependence  Relationships  Among  Multivariate  Exponential  Variables 

Many  multivariate  distributions  have  been  described  and  applied 
to  reliability  problems  [4,  8,  10].  In  each  case  the  marginal  uni- 
variate distributions  are  of  the  negat  ive  exponential  form.  Properties 
of  the  distribution  used  here  are  most  easily  derived  by  exploiting  its 
relationship  to  multivariate  normal  distributions  [3,  5]. 

The  multivariate  exponential  variables  r^,  . . . , r n can  be 

viewed  as  sums  of  squares: 

ri=wi2+zf'  (A-U 

where  w and  z_  are  independent,  zero  mean,  identically  distributed  nor- 
mal vectors,  each  with  covariance  matrix  F.  It  follows  that  the  r^  have 
exponential  marginal  distributions  with 


E[r.] 


2 


7ii 


P 


r. 

i 


r. 

J 


(A-2) 


We  specialize  to  the  case  where  the  underlying  normal  sequences  {w^} 
and  {z.}  are  Markovian 


* V iru  Yjj 


HI 


(A-3) 


and  find  that  {r^}  is  also  Markov  with  the  joint:'  density 


I 


25- 


that  the  following  determinant*  are  non-negative  for  any  N and  any 
*1  < a2  < aN:  ^ < 02  **•  < 0N’ 


f(®j • 0 2^*  * * ^ 


f(®jsj*  ) ftojsj* 


> 0 


THEOREM:  If  h(y)  is  continuous  and  convex,  and  satisfies  the  bounds 

i)  h(0)  < 0 

ii)  | h(y)|  < a + b y2m;  a > 0,  b > 0,  y > 0,  m » positive  integer. 
Then  g(x)  is  continuous,  convex,  bounded  in  the  sense 

|g(x)|  < a'  +b'  x2m;  a’  >0,  b'  >0,  x>0: 
and  belongs  to  one  of  the  three  following  categories: 

(I)  g(x)  > 0 for  all  x > 0, 

(II)  g(x)  jc  0 for  all  x > 0 except  with  a possible  zero  at  x * 0, 

j}t  ^ 

(III)  there  exists  a unique  x , 0 < x < such  that  g(x)  > 0 for 

$ >9* 

all  x >x  : and  g(x)  < 0 for  x < x except  for  a possible  zero 
at  x * 0. 

This  theorem  is  used  to  define  optimal  decision  regions  according  to 
the  sign  of  a function  like  g(x),  with  x corresponding  to  a decision 
threshold. 


